上一篇,我們根據 token 內容得知 OIDC provider URL 設定為 EKS OICD provider,然而為什麼 IAM OIDC 與 EKS 關係為何?其 token 又是怎麼被掛載注入於 Pod 呢?
而根據 EKS IRSA [1] 文件內提及以下內容:
AssumeRoleWithWebIdentity
[2] API ,並取得 IAM temporary security credentials,並藉由此 credentials 與 AWS 服務互動。ProjectedServiceAccountToken
功能。此功能是 OIDC JSON Web Token,同時包含服務帳戶身分,並支援可設定的對象。故本文接續討論:
首先,OpenID Providers 必須提供提供一個 JSON 格式文件設定檔於路徑 /.well-known/openid-configuration
給 issuer。因此我們可以透過 curl
命令檢視 EKS OIDC endpoint 所提供設定,其詳細欄位定義查看官方 OIDC 文件欄位定義[3]:
$ curl -s https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/.well-known/openid-configuration | jq .
{
"issuer": "https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"jwks_uri": "https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/keys",
"authorization_endpoint": "urn:kubernetes:programmatic_authorization",
"response_types_supported": [
"id_token"
],
"subject_types_supported": [
"public"
],
"claims_supported": [
"sub",
"iss"
],
"id_token_signing_alg_values_supported": [
"RS256"
]
}
其中我們關注於 jwks_uri
:提供 JSON Web Key (JWK) set 的 openid-configuration 中指定的 endpoint,換言之,主要目的用於簽署來自 issuer 的 token。
$ curl -s https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/keys | jq .
{
"keys": [
{
"kty": "RSA",
"e": "AQAB",
"use": "sig",
"kid": "001bbf00f107017bc195419a1f47c4d3debad7d8",
"alg": "RS256",
"n": "qxowTumR7IVAaxNmMluNDliaZO-rRGnREr0s7oIFZf2iJzO31UW7W8a3MGOauHxrjMxhop0yO4ScKfP29OsCOAoG--_0mbVUnNYaa4aFbB5QqFhynXYGuITTi4AuRs4nMxdlViMA_AloC57q_dKGnfW2azdKpr8hFWv6y_NX_oauiJ1LVQdhpHXi5WJkev8yxsUH5L3Nzs3016j3t8VEQ6CjUt71BBcpeSLAOiEPM99EnNdF9oUmHttnpg09laK7dEuR0LkwVJd_MuNkAh8qn9GWl-sJOGqkvwCGu7i_p92BvvB3Uf0WdQRqadWBuWcdCR69Khp0DdHhhEv2blAzuw"
},
{
"kty": "RSA",
"e": "AQAB",
"use": "sig",
"kid": "1e1d3732da4f678d29629aa8904535c354a50cc9",
"alg": "RS256",
"n": "s9nXfKxBjAY3akEnV5Enr76ZLkXCoeWjm0Ydr5bhubEAhWGszrPaLFVGskiddlYnGa_GGRDFFs-Lr_Gvb_Mxcvk0BfuylhAdfW6_slq6mmEJqw_HIekYnWmd2B6JIyGgmOFtebWEyKWgQd_U_WxPRxUqTh-wnEDP8Lb7V5F15lhGYtZzxKQMLxIDfbZVJmjQ2iruIUAMMXTgGMOSIq6dg4f4YE2-VnTr3_e35NHY83A0fr4_LnyMq7LX9LC5mBWH5yU4GFUvBo39eOMopZSBnEoytzNbHVNwUr7HFSRpKtXsUQCo5VqSRF-wq8cxT7FGpeETBoBGO5gdbvGjpCZJ-Q"
}
]
}
AWS STS 藉由在 jwks_uri
定義的 endpoint https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/keys 檢查此 token 已經由信任來源(Identity Provider defined in IAM,如 EKS)簽署並驗證其 signature。以下整理 OIDC 流程:
kid
(key ID)attribute 的 public key。 如果它不識別 key,則會有以下流程:
/.well-known/openid-configuration
路徑取得 OIDC 設定檔。jwks_uri
欄位取得 provider 的 published public keys (the JWKS)。在 IAM 中建立了 OIDC Provider 後,我們可以在 AWS IAM 上設定為信任由它產生的 token,因此需要於 IAM role trust policy [4]上達到這件事。而在昨天使用了 eksctl create iamserviceaccount
命令建立 Kubernetes Service Account,我們可以檢視一下此 IAM role 建立過程中是否有調整對應的 IAM role trust policy。
透過 aws iam get-role
命令檢視 Role.AssumeRolePolicyDocument
。在此範例中:我們授權(authorize)client 使用帶有 system:serviceaccount:default:my-s3-sa
的 sub
聲明 JWT token 進行 assumed role,其中此為 default
namespace 中 Service Account my-s3-sa
。
$ aws iam get-role --role-name eksctl-ironman-2022-addon-iamserviceaccount-defau-Role1-KXGCZM9EIHAA --query Role.AssumeRolePolicyDocument
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:aud": "sts.amazonaws.com",
"oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:default:my-s3-sa"
}
}
}
]
}
在了解 OIDC 流程之後,所以提供驗證的 token 到底是怎麼來的。讓我們再次複習一下 Pod 資訊:
$ kubectl describe po aws-cli
Name: aws-cli
Namespace: default
...
...
Containers:
aws-cli:
...
Command:
sleep
infinity
State: Running
Started: Fri, 23 Sep 2022 10:59:13 +0000
Ready: True
Restart Count: 0
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: eu-west-1
AWS_REGION: eu-west-1
AWS_ROLE_ARN: arn:aws:iam::111111111111:role/eksctl-ironman-2022-addon-iamserviceaccount-defau-Role1-KXGCZM9EIHAA
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cvtwf (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
kube-api-access-cvtwf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
其中有兩筆 Projected Volumes[5]:
kube-api-access-cvtwf
掛載於 /var/run/secrets/kubernetes.io/serviceaccount
目錄。此為 ServiceAccount Admission Controller [6] 更新 Pod Spec,倘若 Pod Spec 沒有指定 Service Account 則會設定該 namespace 中 default Service Account。 而在 Kubernetes 1.22 版本正式啟用 BoundServiceAccountTokenVolume
[7],改善了 Service Account Token 安全性問題,並設定過期時間為一小時。aws-iam-token
掛載於 /var/run/secrets/eks.amazonaws.com/serviceaccount
目錄。此由 Amazon EKS Pod Identity Webhook[8] 定義的 projected volume,以下為部分原始碼[9]定義:// NewModifier returns a Modifier with default values
func NewModifier(opts ...ModifierOpt) *Modifier {
mod := &Modifier{
AnnotationDomain: "eks.amazonaws.com",
MountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount",
volName: "aws-iam-token",
tokenName: "token",
}
for _, opt := range opts {
opt(mod)
}
return mod
}
接續,為瞭解 EKS 於 API server 上是否有設定相對應 Service Account 設定。透過 CloudWatch Logs insight syntax 語法檢視 kube-apiserver
logs 內啟用與 Service Account 所關聯的 flag:
filter @logStream not like /^kube-apiserver-audit/
| filter @logStream like /^kube-apiserver-/
| fields @timestamp, @message
| sort @timestamp asc
| filter @message like "--service-account"
| limit 10000
I0914 09:46:06.296376 10 flags.go:59] FLAG: --service-account-api-audiences="[https://kubernetes.default.svc]"
I0914 09:46:06.296386 10 flags.go:59] FLAG: --service-account-extend-token-expiration="true"
I0914 09:46:06.296391 10 flags.go:59] FLAG: --service-account-issuer="[https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]"
I0914 09:46:06.296402 10 flags.go:59] FLAG: --service-account-jwks-uri=""
I0914 09:46:06.296407 10 flags.go:59] FLAG: --service-account-key-file="[/etc/kubernetes/pki/sa.pub]"
I0914 09:46:06.296417 10 flags.go:59] FLAG: --service-account-lookup="true"
I0914 09:46:06.296422 10 flags.go:59] FLAG: --service-account-max-token-expiration="24h0m0s"
I0914 09:46:06.296428 10 flags.go:59] FLAG: --service-account-signing-key-file=""
其中我們關注以下三個 flag:
--service-account-issuer
: Service Account issuer 名稱,可以觀察到預設定義為 EKS OIDC Provider URL。--service-account-key-file
:用來驗證 ServiceAccount tokens key,設置為 /etc/kubernetes/pki/sa.pub
。代表了 EKS 在建立 cluster 時預設建立此 key pair。--service-account-api-audiences
:設置 API audiences 為 https://kubernetes.default.svc
。當 AWS API 被呼叫時,AWS SDK 呼叫 STS AssumeRoleWithWebIdentity
API。在驗證過 token signature 後,IAM 將 Kubernetes issue 的 token 交換為臨時 AWS role credential。
根據 Java SDK Credential Provider Chain[10] 流程為:
AWS_ACCESS_KEY_ID
及AWS_SECRET_ACCESS_KEY
。aws.accessKeyId
及 aws.secretAccessKey
。~/.aws/credentials
多半由 AWS SDKs 或是 AWS CLI 設定。AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
。其中 Web Identity Token credentials 所提及的環境變數為 AWS_WEB_IDENTITY_TOKEN_FILE
,我們也可以根據不同語言文件來進行確認,如 Java SDK[11] 或 js SDK[12]。
EKS cluster 在建立時,預設設定了 Project Service Account 為 OIDC Provider 及設定對應的 key。而 IAM 服務也支援作為 OIDC provider,故能在 issue token 時驗證授權 token。同時,在 Pod 建立階段透過 Amazon EKS Pod Identity Webhook 更新 token 於環境變數及掛載目錄 。最後,AWS SDK 藉由 STS AssumeRoleWithWebIdentity
API 取得對應 IAM role 暫時 credential 調用對應的 AWS 服務 API。
上述資訊透過 EKS 所提供 Logs 來驗證上游 Kubernetes 運作原理,倘若上述內文有所錯誤,隨時可以留言或是私訊我。